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1.  Introduction 


Every  decade  the  Department  of  Defense  determines  its  technical  challenges  for  the  next 
10  to  20  years.  The  top  Air  Force  (AF)  priorities  identified  in  2010  allocate  a  prominent  role  to 
computational  intelligence.  This  article  reviews  those  priorities  and  discusses  cognitive 
algorithms  resulting  in  breakthroughs  in  defense  applications,  and  making  progress  toward 
achieving  the  AF  goals.  The  mathematical  techniques  of  dynamic  logic  and  neural  modeling 
fields  model  aspects  of  the  functionality  of  the  mind:  perception,  cognition,  and  higher  cognitive 
functions.  This  leads  to  a  significant  improvement  in  classical  signal  processing  applications, 
including  detection,  tracking,  and  fusion  in  noise  and  clutter,  making  autonomous  situational 
awareness  possible.  The  first  part  of  this  paper  describes  past  difficulties,  why  these  problems 
have  not  been  solved  for  decades,  and  the  gist  of  the  current  approaches.  The  second  part  of  this 
paper  addresses  future  research  directions:  the  augmentation  of  human  performance  through 
improvements  in  man-machine  interfaces.  This  includes  modeling  of  language  interacting  with 
cognition.  In  future  systems,  computers  will  learn  from  humans,  and  humans  will  operate 
machines  by  language  and  thoughts. 

The  2010  report  “Technical  Horizons”  by  the  Chief  Scientist  of  the  US  Air  Force 
analyzes  many  details  of  the  technical  challenges  faced  by  the  Air  Force  over  the  next  20  years 
[1].  Among  three  most  essential  focus  areas,  the  first  two  identify  a  need  for  autonomous 
systems  and  augmentation  of  human  performance.  The  most  important  technologies  include  the 
need  for  processing  of  the  overwhelming  amount  of  signals  collected  by  diverse  sensors. 

The  brain  works  better  than  computers.  Therefore,  computational  intelligence  attempts  to 
model  brain  mechanisms  and  to  apply  these  models  to  developing  cognitive  algorithms,  which 
would  bring  computer  system  performance  closer  to  the  performance  of  the  brain-mind  [2]. 
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These  attempts  began  in  the  1950s,  and  the  first  designers  of  computational  intelligence  were 
sure  that  soon  computers  would  overtake  human  intelligence.  Yet  modeling  the  brain-mind 
turned  out  to  be  much  more  challenging  than  expected.  Computers  still  cannot  do  what  is  easily 
done  by  children  and  animals.  The  Chief  Scientist  of  the  Air  Force  has  identified  the  improving 
of  computer  abilities  as  the  most  important  technical  horizon  for  USAF  research  over  the  time 
span  2010-30.  Section  II  discusses  specific  mathematical  reasons,  which  made  computational 
intelligence  a  far  more  difficult  problem  than  originally  expected.  We  also  touch  on  fundamental 
psychological  mechanisms  that  have  diverted  outstanding  scientists  from  looking  in  promising 
directions  to  solve  these  problems. 

Experimental  neuroimaging  and  cognitive  neuroscience  discovered  several  fundamental 
principles  of  the  brain-mind  organization.  Among  these  are  interactions  between  bottom-up  and 
top-down  signals  [3, 4, 5, 6],  a  theory  of  instincts  and  emotions  [7],  the  knowledge  instinct  and 
aesthetic  emotions  [8,9,10,1 1,12,13]  extending  the  previous  reference  [7]  to  higher  cognitive 
functions,  and  to  mechanisms  of  “vague-to  crisp”  transformation  of  initially  vague  and 
unconscious  states  of  top-down  projections  [14].  Section  III  describes  these  experimental 
findings,  mathematical  models  of  these  processes,  dynamic  logic  (DL),  neural  modeling  fields 
(NMF),  and  their  fundamental  role  in  overcoming  past  difficulties  as  discussed  in  the  previous 
section. 

Section  IV  describes  applications  of  DL-NMF  to  classical  signal  processing  problems, 
detection,  tracking,  and  fusion  of  sensor  signals  from  multiple  platforms  in  strong  clutter.  These 
problems  have  not  been  solved  for  decades.  Also  autonomous  learning  of  situations  is  addressed, 
which  is  a  problem  of  the  next  level  of  complexity.  Section  V  addresses  an  emerging  engineering 
problem  of  learning  language,  and  interaction  between  language  and  cognition.  This  is  crucial 
for  the  augmentation  of  human  performance  through  the  improvement  of  the  human-computer 
interactions.  Section  VI  discusses  future  research  and  development  directions. 
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2.  Past  difficulties:  complexity  and  logic 


The  perception  and  cognition  abilities  of  computers  still  cannot  compete  with  those  of 
kids  and  animals  [15].  Most  algorithms  and  neural  networks  suggested  since  the  1950s  for 
modeling  perception  and  cognition,  as  discussed  in  [16,17,18,19],  faced  the  difficulty  of 
combinatorial  complexity  (CC).  Rule  systems  of  artificial  intelligence  in  the  presence  of 
variability  have  grown  in  complexity:  rules  have  become  contingent  on  other  rules,  and  rule 
systems  faced  CC.  Learning  algorithms  and  neural  networks  have  to  be  trained  to  understand  not 
only  individual  objects,  but  also  combinations  of  objects,  and  thus,  faced  CC  of  training.  Fuzzy 
systems  required  a  fuzziness  level  to  be  set  appropriately  in  different  parts  of  the  systems,  also 
degrees  of  fuzziness  vary  in  time,  and  attempts  to  select  efficient  levels  of  fuzziness  would  lead 
toCC. 

These  CC  difficulties  were  related  to  Godelian  limitations  of  logic;  they  were 
manifestations  of  logic  inconsistency  in  finite  systems  [8,20].  Even  approaches  designed 
specifically  to  overcome  logic  limitations,  such  as  fuzzy  logic  and  neural  networks,  encountered 
logical  steps  in  their  operations:  neural  networks  are  trained  using  logical  procedures  (e.g.  “this 
is  a  chair”),  and  fuzzy  systems  required  logical  selection  of  the  degree  of  fuzziness. 

Millennia  logic  was  associated  with  the  essence  of  mind  mechanisms.  Near  the  beginning 
of  the  20th  century,  Hilbert  was  sure  that  his  logical  theory  also  described  the  mechanisms  of  the 
mind:  “The  fundamental  idea  of  my  proof  theory  is  none  other  than  to  describe  the  activity  of  our 
understanding,  to  make  a  protocol  of  the  rules  according  to  which  our  thinking  actually 
proceeds”  [21].  This  vision  of  logic  modeling  the  mind  was  shattered  by  Godel’s  discoveries  in 
the  1930s  [22].  But  why  25  years  after  Godel,  in  the  1950s,  where  the  founders  of  artificial 
intelligence  sure  that  logic  is  adequate  for  modeling  of  the  mind? 
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The  reason  for  the  fascination  by  logic  is  fundamental  to  science,  engineering,  and  related 
to  the  mechanisms  of  the  mind  was  only  understood  recently  [4,8,14,23],  Most  of  the  mind’s 
mechanisms  are  usually  inaccessible  to  consciousness,  e.g.,  we  are  not  conscious  about  the 
individual  neural  firings  and  intermediate  signal  processing  steps.  Only  the  “final  results”  of 
perception  and  cognition,  nearly  crisp  logic-like  perceptions  and  thoughts,  are  available  to 
consciousness.  These  “final  results”  approximately  obey  the  rules  of  logic.  Logical  conscious 
states  are  like  islands  in  the  ocean  of  unconsciousness.  But  in  our  consciousness  there  are  only 
crisp  logical  results,  and  consciousness  works  so  that,  while  jumping  through  oceans  of 
unconsciousness,  we  subjectively  feel  as  if  we  smoothly  flow  from  one  conscious  logical  state  to 
the  next.  Our  intuitions  about  the  mind,  including  scientific  intuitions  are  strongly  biased  towards 
logic.  This  is  why,  most  algorithms  and  neural  networks  (even  if  purposefully  designed  to 
oppose  logic)  use  logic  in  a  fundamental  way.  Due  to  scientific  findings  we  know  today  that 
logic  is  not  a  fundamental  mechanism  of  the  mind. 

3.  Cognitive  mechanisms:  mathematical  models  and  experimental  evidence 

Cognitive,  neurological  and  neuroimaging  studies  [6,8,14]  demonstrated  that  the  mind 
understands  the  world  by  modeling  it.  Here  the  word  “model”  is  used  in  two  distinct  meanings. 
First,  the  mind  models  the  world;  second,  the  article  describes  mathematical  processes  that 
model  these  mechanisms  of  the  mind.  So  we  talk  about  mental  models  of  the  world  and 
mathematical  models  of  the  mind.  The  mind  models  the  world  using  mental  representations,  or 
models  of  the  world.  Consider  a  simplified  process  of  visual  perception.  To  perceive  an  object,  a 
mental  representation  of  this  object  in  our  memory  should  be  matched  to  patterns  in  sensor 
signals  corresponding  to  the  object.  During  this  matching  process,  the  retina  projects  sensor 
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signals  onto  the  visual  cortex;  these  projections  are  called  bottom-up  signals.  In  parallel,  object 
representations  in  memory  (mental  models)  send  top-down  signals  (projections)  to  the  visual 
cortex  [4,8,14],  The  mind  is  organized  in  a  multi-level,  approximately-hierarchical  structure. 
Higher  level  cognition  involves  higher  levels;  matching  bottom-up  and  top-down  signals 
between  adjacent  levels  is  the  mechanism  of  cognition. 

Neuroimaging  studies  [14]  have  demonstrated  that  conscious  perceptions  are  preceded  by 
activation  of  cortex  areas,  where  the  top-down  signals  originate.  Experiments  also  demonstrated 
that  the  initial  top-down  projections  are  vague  and  unconscious  (or  less  conscious)  than 
perceptions  of  objects  at  the  end  of  the  matching  process.  As  discussed  in  this  article,  these 
mechanisms  are  fundamental  for  the  functioning  of  the  mind;  they  enable  the  mind  to  overcome 
difficulties  of  the  mathematical  models  discussed  in  the  previous  sections  [8,24,25,26].  The 
mathematical  models  of  the  mind  mechanisms  help  in  understanding  properties  of  the  mind  that 
previously  seemed  mysterious.  They  also  lead  to  cognitive  algorithms,  which  significantly 
outperform  the  previous  state  of  the  art  [8],  Mathematical  models  of  these  processes,  dynamic 
logic  (DL)  and  neural  modeling  fields  (NMF)  are  discussed  below. 

NMF  mathematically  describes  multi-level,  approximately-hierarchical  structure  of  the 
mind  [8]  discussed  above.  At  each  level  in  NMF,  there  are  mathematical  models  of  mental 
representations,  M(m)  (called  models,  m  =  1,...  M)  and  mathematical  description  of  the 
matching  process.  Representations,  M(m),  encapsulate  the  mind’s  knowledge.  As  discussed,  they 
generate  top-down  neural  signals,  interacting  with  bottom-up  signals,  X(n).  Models  M(m) 
predict  patterns  in  signals  X(n),  which  correspond  to  object  m.  They  depend  on  parameters,  Sm, 
characterizing  a  particular  view  of  the  object  m  (distance,  angles,  lightings,  etc.).  Interactions 
between  signals  are  governed  by  the  knowledge  instinct,  which  drives  the  matching  of  bottom-up 
and  top-down  signals.  In  this  matching  process,  multiple  vague  models  compete  with  each  other 
and  are  modified  for  better  matching  patterns  in  bottom-up  signals.  This  constitutes  learning, 
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adaptation,  and  if  needed,  formation  of  new  models  for  better  correspondence  to  the  input 
signals. 

A  mathematical  model  of  the  knowledge  instinct  is  described  as  maximization  of  a  similarity 
measure,  L,  between  bottom-up  and  top-down  signals,  [8,17]. 

L=n  l  (X(n»;  l  (X(n»  =  £  r(m)i(X(n)|M(m».  (1) 

neN  m<=M 

Here,  t  (X(n)  |  M(m))  or  l  (n|m)  for  shortness  is  a  similarity  of  signal,  X(n),  “conditional”  on 
object  m  being  present,  [8].  Therefore,  when  combining  these  quantities  into  the  overall 
similarity  measure,  L,  they  are  multiplied  by  rates  r(m),  which  represent  a  probabilistic  measure 
of  object  m  actually  being  present;  rates,  r(m),  are  unknown  parameters  along  with  Sm. 

The  learning,  or  matching  bottom-up  and  top-down  signals,  consists  of  associating  signals,  n, 
with  concepts,  m,  and  estimating  model  parameters,  S,  by  maximizing  similarity  L.  Note  that  all 
possible  combinations  of  signals  and  models  are  accounted  for  in  similarity  (1).  This  can  be  seen 
by  expanding  a  sum  in  (1),  and  multiplying  all  the  terms  which  would  result  in  a  huge  number  of 
Mn  items.  This  is  the  number  of  combinations  between  all  signals,  N,  and  all  models,  M.  Here  is 
the  source  of  CC  of  many  algorithms  used  in  the  past.  NMF  solves  this  problem  by  using 
dynamic  logic  (DL)  [17,20].  An  important  aspect  of  DL  is  matching  vagueness  or  fuzziness  of 
similarity  measures  to  the  uncertainty  of  models.  Initially,  parameter  values  are  not  known,  and 
uncertainty  of  the  models  is  high;  so  is  the  fuzziness  of  the  similarity  measures.  In  the  process  of 
learning,  the  models  become  more  accurate,  the  similarity  measure  more  crisp  and  the  value  of 
the  similarity  increases.  This  is  the  mechanism  of  DL. 

Mathematically  it  is  described  as  follows.  First,  assign  any  values  to  unknown 
parameters,  Sm.  Then,  compute  association  variables  f(m|n), 
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(2) 


f(m|n)  =  r(m)  t  (n|m)  /  £  r(m')  ^(X(n)|m’). 

m'eM 


DL  is  defined  as  follows, 


dSm/dt  =  X  f(m | n) [otrV(n | m)/ oM m]5M J <3Sm , 

neN 


(3) 


Parameter  t  is  the  time  of  the  internal  dynamics  of  the  NMF  system,  like  a  number  of  iterations, 
if  solving  (3)  by  iterative  steps.  On  every  iteration,  new  values  of  parameters  are  substituted  in 
eq.(2).  The  above  procedure  has  been  proven  to  converge  [8].  Local  maxima  are  avoided  due  to 
DL  beginning  with  vague  similarity  measures,  which  smooth  local  maxima  over  [25,27,23].  This 
process  “from  vague  to  crisp”  is  the  essence  of  DL.  Various  possible  neural  mechanisms  of  DL 
in  the  brain  are  discussed  in  [28,29,30], 

4.  Applications  of  DL-NMF  to  classical  previously  unsolved  problems 

NMF-DL  was  applied  to  a  number  of  complicated  problems  fundamental  for  defense 
applications  in  signal  processing,  detection,  and  clustering  problems  [8,9,23,27,31,32,33,34,35, 
36,37,38],  which  could  not  have  been  solved  previously  because  of  strong  clutter  interfering  with 
signals.  Obtained  solutions  often  approach  information-theoretic  performance  bounds  and 
therefore  are  the  best  possible  [39,40,41,42,43].  Here,  we  briefly  illustrate  three  problems: 
detection,  tracking,  fusion  in  strong  clutter  [44,45,46,47,48].  We  also  describe  a  solution  to  a 
longstanding  unsolved  problem  of  autonomous  situational  awareness. 
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In  the  first  example,  NMF-DL  is  detecting  ‘smile’  and  ‘frown’  patterns  in  noise  shown  in 
Fig.  la  without  clutter,  and  in  Fig.  lb  with  clutter,  as  actually  measured  [233].  This  example  is 
beyond  the  capabilities  of  previously  existing  techniques  because  of  the  computational 
complexity.  The  DL  complexity  in  this  example  is  equal  109,  so  that  a  problem,  previously 
unsolvable  due  to  complexity,  has  been  solved  using  NMF-DL. 


Fig.l.  An  example  of  NMF-DL  perception  of  ‘smile’  and  ‘frown’  objects  in  clutter  in  2-dimensional  space:  (a)  true 
‘smile’  and  ‘frown’  patterns  are  shown  without  clutter;  (b)  actual  image  available  for  recognition  (signal  is  below 
clutter,  S/C  ~  0.5);  (c)  an  initial  fuzzy  blob-model,  the  fuzziness  corresponds  to  uncertainty  of  knowledge;  (d) 
through  (h)  show  improved  models  at  various  iteration  stages  (total  of  22  iterations).  The  improvement  over  the 
previous  state  of  the  art  is  7,000%  in  S/C. 
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e)  5  iterations  f)  9  iterations  g)  12  iterations  h)  converged  state 


Fig.  2.  Joint  detection  and  tracking  of  objects  below  clutter  using  NMF-DL:  (a)  true  track  positions  in  1  km  x  1  km 
data  set;  (b)  actual  data  available  for  detection  and  tracking.  Evolution  of  the  NMF-DL  is  illustrated  in  (c)-(h), 
where  (c)  shows  the  initial,  uncertain  model  and  (h)  shows  the  model  upon  convergence  after  20  iterations. 
Converged  models  are  close  to  the  truth  (a). 

The  second  example,  Fig.  2,  shows  NMF-DL  joint  detection  and  tracking  of  targets 
below  clutter.  The  third  axis,  time,  is  perpendicular  to  the  page;  6  time  scans  are  shown  collapsed 
on  top  of  each  other,  details  of  this  example  are  discussed  in  [27].  In  this  example,  targets  cannot 
be  detected  on  a  single  scan,  therefore  detection  and  tracking  have  to  be  performed 
simultaneously  (“track  before  detect”);  in  terms  of  S/C  ratio  the  improvement  is  approximately 
8,000%. 

The  third  example,  Fig. 3,  shows  NMF-DL  joint  detection,  tracking,  and  fusion  of  sensors 
from  three  platforms  [38],  Fig. 3  shows  1  scan  from  each  sensor  and  several  DL  iterations.  In  this 
case,  because  of  low  signal  to  clutter  ratio,  S/C,  targets  cannot  be  detected  from  a  single  sensor, 
therefore,  joint  fusion,  tracking,  and  detection  have  to  be  perfonned  concurrently.  In  addition, 
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GPS  does  not  give  enough  accuracy  for  triangulation,  therefore,  relative  target  localization  has  to 
be  perfonned  concurrently  with  other  processing.  Problems  of  this  complexity  have  not  been 
previously  solved. 


Fig. 3.1,  sensor  1  Fig. 3.2,  sensor  2  Fig. 3. 3,  sensor  3 

Fig. 3.  Illustrates  NMF-DL  joint  detection,  tracking,  and  fusion  of  sensors  from  3  platforms.  The  figure  shows  for 
each  sensor:  four  true  tracks,  1  scan,  where  targets  are  not  seen  under  clutter,  and  several  iterations  of  DL.  In  this 
case  targets  cannot  be  detected  from  a  single  sensor,  therefore  joint  fusion,  tracking,  and  detection  have  to  be 
performed  concurrently.  In  addition,  GPS  does  not  give  enough  accuracy  for  triangulation,  therefore,  relative  target 
localization  has  to  be  performed  concurrently  with  other  processing. 


The  fourth  example,  Figs.  4  through  6,  illustrates  autonomous  learning  of  situations,  a 
problem  of  the  next  level  of  complexity  comparative  to  the  previously  considered.  Autonomous 
learning  of  situations  was  recognized  as  an  important  military  problem  for  decades,  however  it 
was  not  solved.  It  is  difficult  because  every  situation  contains  several  objects  relevant  to  this 
particular  situation,  and  a  large  number  of  irrelevant  objects.  The  data  are  shown  in  Fig.  4. 
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Fig.  4.  Learning  situations;  white  dots  show  present  objects  and  black  dots  correspond  to  absent  objects.  Vertical 
axes  show  1000  objects,  horizontal  axes  show  10  situations  each  containing  10  relevant  objects  and  40  random 
ones;  in  addition  there  5000  “clutter”  situations  containing  only  random  objects.  Fig.  4a  shows  situations  sorted 
along  horizontal  axis,  hence  there  horizontal  lines  corresponding  to  relevant  objects  (the  right  half  contains  only 
random  noise).  Fig  4b  shows  the  same  situations  in  random  order,  which  looks  like  random  noise. 

In  real  life,  situations  are  often  coming  in  random  order,  without  a  teacher  pointing  to  a 
particular  object  indicating  situations  like  in  Fig.  4b.  To  find  situations,  say  by  sorting  along 
horizontal  axes,  until  horizontal  lines  appear  as  in  Fig.  4a,  would  take  about  MN  operations; 
assuming,  say,  M=20  situations,  the  number  of  operations  is  ~  lO13000.  NMF-DL  solves  the 
problem  as  described  in  [49,50].  The  solution  and  the  low  level  of  errors  are  illustrated  in  Fig.  5. 
Fig.  5a  illustrates  DL  iterations  beginning  with  random  association  of  objects  and  (arbitrary 
taken)  20  situations.  Fig.  5b  illustrate  that  errors  quickly  go  to  a  small  value.  After  the  problem 
of  associating  objects  and  situations  are  solved  by  using  DL,  the  correct  situations  are  chosen  by 
matching  to  the  known  data.  The  error  does  not  go  to  0  for  numerical  reasons  as  discussed  in 
[50]. 


11 


it  =  0  it  =  1  it  =  2  it  =  3 


—  1 

1 

.  \ 

~~'-0 - * 

- -o - » - -o- - \ 

a  b 

Figure  5.  (a)  shows  DL  initiation  (random)  and  the  first  three  iterations  (Fig.  5a);  the  vertical  axis  shows  objects  and 
the  horizontal  axis  shows  models  (from  1  to  20).  The  problem  is  approximately  solved  by  the  third  iteration.  This  is 
illustrated  in  Fig  5b,  where  the  error  is  shown  on  the  vertical  error.  The  correct  situations  are  chosen  by  minimizing 
the  error.  The  error  does  not  go  to  0  for  numerical  reasons  as  discussed  in  [50]. 

In  the  above  example,  relationships  (such  as  on-the-left-of,  or  under)  have  not  been 
explicitly  considered.  They  can  be  easily  included.  Every  relation  and  object  can  include  a 
marker,  pointing  what  relates  to  what.  These  markers  are  learned  the  same  way  as  objects 
[49,50]. 

Solving  problems  like  detection,  tracking,  and  fusion  require  models  matching  the 
problem,  which  is  described  in  the  given  references.  The  problem  of  learning  situations  is  a 
general  one,  it  is  appropriate  for  matching  the  bottom-up  and  top-down  signals  at  every  level  in 
the  approximate  hierarchy  of  the  mind.  Following  sections  describe  how  it  can  be  applied  to 
solving  new  emerging  class  of  engineering  problems,  including  interaction  between  language 
and  cognition. 
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5.  Cognition  and  language 


As  discussed,  augmentation  of  human  performance  by  computers  has  been  identified  by 
the  AF  as  a  top  priority.  This  goal  faces  the  bottleneck  of  the  human-machine  interface  [51,52]. 
Overcoming  this  bottleneck  is  possible  by  mathematically  modeling  interaction  between 
language  and  cognition  [53,54,24].  Before  attempting  mathematical  modeling  of  language,  one 
needs  to  understand  why  previous  decades  of  research  in  this  direction  have  not  succeeded. 
Therefore,  this  section  begins  with  the  mysteries  facing  this  problem  so  far,  and  how  cognitive 
computational  intelligence  building  on  NMF-DL  promises  to  resolve  them.  These  mysteries 
include  the  following:  In  what  way  is  language  and  cognition  similar  and  different?  Do  we  think 
with  words?  Or  do  we  use  words  for  communicating  completed  thoughts?  Why  children  learn 
language  by  5  years  of  age  and  can  talk  practically  about  the  entire  contents  of  culture,  but 
cannot  act  like  adults?  Why  are  there  no  animals  thinking  like  humans  and  speaking  no  human¬ 
like  language? 

Consider  first  how  is  it  possible  to  leam  which  words  correspond  to  which  objects? 
Contemporary  psycholinguists  follow  the  ancient  Locke  idea,  Associationsim:  associations 
between  words  and  object  are  just  remembered.  But  this  is  mathematically  impossible.  The 
number  of  combinations  among  100  words  and  100  objects  is  larger  than  all  the  elementary 
particle  interactions  in  the  Universe.  Combinations  of  30,000  words  and  objects  are  practically 
infinite.  Mathematical  linguists  did  not  offer  any  solution.  NMF-DL  solves  this  problem  using 
the  dual  model  [55,56,57].  Every  mental  representation  consists  of  a  pair  of  two  models,  or  two 
model  aspects,  cognitive  and  language.  Mathematically,  every  concept-model  Mm  has  two  parts, 
linguistic  MLm  and  cognitive  MCm: 

Mm  =  {  MLm,  MCm  };  (4) 
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This  dual-model  equation  suggests  that  the  connection  between  language  and  cognitive  models  is 
inborn.  In  a  newborn’s  mind,  both  types  of  models  are  vague  placeholders  for  future  cognitive 
and  language  contents.  An  image,  say  of  a  chair,  and  the  sound  “chair,”  do  not  exist  in  a 
newborn’s  mind.  But  the  neural  connections  between  the  two  types  of  models  are  inborn; 
therefore  the  brain  does  not  have  to  leam  associations  between  words  and  objects  -  which 
concrete  word  goes  with  which  concrete  object.  Models  acquire  specific  contents  in  the  process 
of  growing  up  and  learning,  linguistic  and  cognitive  contents  are  always  staying  properly 
connected.  Zillions  of  combinations  need  not  be  considered.  The  initial  implementations  of  these 
ideas  lead  to  encouraging  results  [58,59,60,61,62,63]. 

Consider  language  hierarchy  higher  up  from  words,  Fig.  6.  Phrases  are  made  up  from 
words  similar  to  situations  made  up  from  objects.  Because  of  the  linear  structure,  language 
actually  is  simpler  than  situations;  rules  of  syntax  can  be  learned  similar  to  learning  objects  and 
relations  using  markers,  as  described  in  the  previous  section.  The  reason  computers  do  not  speak 
English  used  to  be  the  fundamental  problem  of  combinatorial  complexity.  Now  that  the 
fundamental  problem  is  solved,  learning  language  will  be  solved  in  due  course.  Significant  effort 
will  be  required  to  build  machines  learning  language.  However,  the  principal  difficulty  has  been 
solved  in  the  previous  section.  The  mathematical  model  of  learning  situations,  considered  in  the 
previous  section,  is  similar  to  learning  how  phrases  are  composed  from  words.  Syntax  can  be 
learned  similar  to  relations  between  objects  [49,50,64]. 

The  next  step  beyond  current  mathematical  linguistics  is  modeling  the  interaction 
between  language  and  cognition.  It  is  fundamental  because  cognition  cannot  be  learned  without 
language.  Consider  a  widely-held  belief  that  cognition  can  be  learned  from  experience  in  the 
world.  This  belief  is  naive  and  mathematically  untenable.  The  reason  is  that  abstract  concepts- 
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Fig.  6.  Parallel  hierarchies  of  language  and  cognition  consist  of  lower  level  concepts  (like  situations  consist  of 
objects).  A  set  of  objects  (or  lower  level  concepts)  relevant  to  a  situation  (or  higher  level  concept)  should  be  learned 
among  practically  infinite  number  of  possible  random  subsets  (as  discussed,  larger  than  the  Universe).  No  amount  of 
experience  would  be  sufficient  for  learning  useful  subsets  from  random  ones.  The  previous  section  overcame 
combinatorial  complexity  of  learning,  given  that  the  sufficient  information  is  present.  However,  mathematical 
linguistic  theories  offer  no  explanation  where  this  information  would  come  from. 


representations  consist  of  a  set  of  relevant  bottom-up  signals,  which  should  be  learned  from  a 
practically  infinite  number  of  possible  random  subsets  (as  discussed,  a  number  larger  than 
particles  in  the  Universe).  No  amount  of  experience  would  be  sufficient  for  learning  useful 
subsets  from  random  ones.  The  previous  section  overcame  combinatorial  complexity  of  learning, 
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given  that  the  sufficient  information  is  present.  However,  mathematical  linguistic  theories  offer 
no  explanation  where  this  infonnation  would  come  from. 

NMF-DL  with  dual  model  and  dual  hierarchy  suggests  that  information  is  coming  from 
language.  This  is  the  reason  why  no  animal  without  human-type  language  can  achieve  human- 
level  cognition.  This  is  the  reason  why  humans  leam  language  early  in  life,  but  learning 
cognition  (making  cognitive  representations-models  as  crisp  and  conscious  as  language  ones) 
takes  a  lifetime.  Information  for  learning  language  is  coming  from  the  surrounding  language  at 
all  levels  of  the  hierarchy.  Language  model-representations  exist  in  the  surrounding  language 
‘ready-made.’  Learning  language  is  thus  grounded  in  the  surrounding  language. 

For  this  reason,  language  models  become  less  vague  and  more  specific  by  5  years  of  age, 
and  much  faster  than  the  corresponding  cognitive  models  for  the  reason  that  they  are  acquired 
ready-made  from  the  surrounding  language.  This  is  especially  true  about  the  contents  of  abstract 
models,  which  cannot  be  directly  perceived  by  the  senses,  such  as  “law,”  “abstractness,” 
“rationality,”  etc.  While  language  models  are  acquired  ready-made  from  the  surrounding 
language,  cognitive  models  remain  vague  and  gradually  acquire  more  concrete  contents 
throughout  life  guided  by  experience  and  language.  According  to  the  dual-model,  this  is  an 
important  aspect  of  the  mechanism  of  what  is  colloquially  called  the  “acquiring  experience.” 

Human  learning  of  cognitive  models  continues  throughout  the  lifetime  and  is  guided  by 
language  models.  If  we  imagine  a  familiar  object  with  closed  eyes,  this  imagination  is  not  as 
clear  and  conscious  as  perception  with  opened  eyes.  With  opened  eyes,  it  is  virtually  impossible 
to  remember  imaginations.  Language  plays  the  role  of  eyes  for  abstract  thoughts.  On  one  hand, 
abstract  thoughts  are  only  possible  due  to  language,  on  the  other,  language  “blinds”  our  mind  to 
the  vagueness  of  abstract  thoughts.  Whenever  one  talks  about  an  abstract  topic,  he  (or  she)  might 
think  that  the  thought  is  clear  and  conscious  in  his  (or  her)  mind.  But  the  above  discussion 
suggests  that  we  are  conscious  about  the  language  models  of  the  dual  hierarchy.  The  cognitive 
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models  in  most  cases  may  remain  vague  and  unconscious.  During  conversations  and  thoughts, 
the  mind  smoothly  glides  among  language  and  cognitive  models,  using  those  that  are  more  crisp 
and  conscious  -  ‘more  available.’  Scientists,  engineers,  and  creative  people  in  general  are  trained 
to  differentiate  between  their  own  thoughts  and  what  they  read  in  a  book  or  paper,  but  usually 
people  do  not  consciously  notice  if  they  use  representations  acquired  from  personal  experiences 
and  deep  thoughts,  or  from  what  they  have  read  or  heard  from  teachers  or  peers.  The  higher  up 
in  the  hierarchy  the  more  vague  the  contents  of  abstract  cognitive  representations  are,  while  due 
to  the  crispness  of  language  models  we  may  remain  convinced  that  these  are  our  own  clear 
conscious  thoughts. 

Another  mystery  of  human-cognition,  which  is  not  addressed  by  current  mathematical 
linguistics,  is  basic  human  irrationality.  This  has  been  widely  discussed  and  experimentally 
demonstrated  after  discoveries  of  Tversky  and  Kahneman  [65],  leading  to  the  2002  Nobel  Prize, 
According  to  NMF-DL,  it  originates  from  the  discussed  dichotomy  between  cognition  and 
language.  Language  is  crisp  in  the  human  brain,  while  cognition  might  be  vague.  Yet,  collective 
wisdom  accumulated  in  language  may  not  be  properly  adapted  to  one’s  personal  circumstances, 
and  therefore  be  irrational  in  a  concrete  situation.  In  the  12th  century  Maimonides  wrote  that 
Adam  was  expelled  from  paradise  because  he  refused  original  thinking  using  his  own  cognitive 
models,  but  ate  from  the  tree  of  knowledge  and  acquired  collective  wisdom  of  language  [66]. 

The  dual-model  also  suggests  that  the  inborn  neural  connection  between  cognitive  brain 
modules  and  language  brain  modules  (evolving  over  thousands  or  millions  of  years  of  evolution) 
is  sufficient  to  set  humans  on  an  evolutionary  path  separating  us  from  the  animal  kingdom. 
Neural  connections  between  these  parts  of  cortex  existed  for  millions  of  years  ago  due  to  mirror 
neuron  system,  what  Arbib  called  the  “language  prewired  brain’’  [67]. 

The  combination  of  NMF-DL  and  the  dual  model  and  hierarchy  introduces  new 
mechanisms  of  language  and  its  interaction  with  cognition.  These  mechanisms  suggest  solutions 
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to  a  number  of  psycholinguistic  mysteries,  which  have  not  been  addressed  by  mathematical 
linguistics.  These  include  fundamental  cognitive  interaction  between  cognition  and  language, 
similarities  and  differences  between  these  two  mechanisms;  word-object  associations;  why 
children  learn  language  early  in  life,  but  cognition  is  acquired  much  later;  why  animals  without 
human  language  cannot  think  like  humans.  These  mechanisms  connected  language  cognition 
dichotomy  to  ‘irrationality’  of  the  mind  discovered  by  Tversky-Kahneman,  and  to  the  story  of 
the  Fall  and  Original  sin. 

The  mathematical  mechanisms  of  NMF-DL-dual  model  are  relatively  simple  and  follow 
eqs.  (2)  through  (4),  also  see  details  in  the  given  references.  These  mathematical  mechanisms 
correspond  to  the  known  structure  and  experimental  data  about  the  brain-mind.  In  addition  to 
conceptual  mechanisms  of  cognition,  they  also  describe  emotional  mechanisms  and  their 
fundamental  role  in  cognition  and  world  understanding,  including  role  of  aesthetic  emotions, 
beautiful,  sublime,  and  musical  emotions  [68,69,70].  Based  on  these  foundations  future  defense 
applications  of  cultural  models  will  be  developed. 

6.  Conclusions 

Computational  intelligence  is  destined  to  play  the  most  important  role  in  defense 
applications  for  decades  to  come.  Its  role  is  to  augment  human  perfonnance  and  to  automate  as 
much  as  possible  complex  problems  of  analyzing  overwhelming  amounts  of  sensor  data  coming 
from  a  variety  of  sources.  To  achieve  this  role,  cognitive  algorithms  modeling  the  performance 
of  the  human  mind  are  being  developed.  This  requires  the  discovery  and  modeling  the 
fundamental  “first  principles”  of  the  mind  operations.  Several  first  principles  of  the  mind-brain 
organization  have  been  identified  by  a  number  of  researchers  in  the  given  references,  described 
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and  mathematically  modeled  in  this  article.  This  is  leading  to  a  significant  improvement  in 
solving  classical  engineering  problems.  Let  us  summarize  these  mechanisms  here:  instincts, 
emotions,  concepts;  bottom-up  and  top-down  neural  signals  interacting  through  a  vague-to-crisp 
process;  the  knowledge  instincts  and  aesthetic  emotions  governing  this  interaction.  These 
mechanisms  are  shared  with  higher  animals.  Human  level  cognition  requires  human  language. 
The  only  additional  required  mechanism,  it  seems,  is  the  dual  model,  leading  to  the  dual 
hierarchy  of  language  and  cognition.  Future  development  in  this  direction  will  lead  to 
cooperative  man-machine  systems. 

To  achieve  this  and  develop  cooperative  human-machine  defense  applications 
augmenting  human  performance  at  the  level  of  human-level  cognition,  the  mechanism  of  joint 
language  and  cognition  described  above  should  be  implemented  in  a  multi-agent  system,  where 
each  agent  would  possess  the  dual  language-cognition  NMF-DL  model.  Simulations  should 
demonstrate  learning  hierarchical  language  from  human  language,  and  then  learning  hierarchical 
cognition  from  sensor  data,  and  guided  by  language.  The  next  step  will  implement  this  intelligent 
multi-agent  hierarchical  system  in  cooperative  man-machine  systems,  where  computers  will 
learn  from  humans  and  then  augment  human  perfonnance.  The  described  mathematical  approach 
also  will  be  extended  ‘below’  objects  to  creating  object-recognition  systems  where  objects  will 
be  modeled  as  ‘situations’  of  sensory  features.  Similarly,  speech  will  be  modeled  by  linear 
‘situations’  of  sounds. 

Another  approach  to  human  performance  augmentation  by  computers  will  exploit  direct 
connection  between  cognitive  states  of  the  mind  and  computers.  Cognitive  states  of  the  mind  will 
be  sensed  by  EEG  sensor  arrays.  The  difficulty  of  this  approach  is  due  to  noisy  contents  of  EEG 
signals.  New  cognitive  algorithms  will  overcome  this  difficulty  similar  to  Fig.  1,  as  discussed  in 
[71]. 
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An  important  defense  application  is  cultural  intelligence,  which  potentially  will  lead  to 
improved  cultural  understanding,  improved  collaborations  of  diverse  cultures,  reduced  conflicts 
and  casualties.  Extending  the  models  of  language,  cognition,  and  emotions  to  developing 
predictive  cultural  models  might  be  among  the  most  important  emerging  engineering  directions 
in  the  21st  century.  Initial  models  developed  in  [72,73,74,75]  promise  explanation  and  prediction 
of  cultural  phenomena  currently  threatening  world  peace.  Improved  models  will  help  better 
intercultural  understanding  and  achieving  peace  in  the  current  complicated  multi-cultural  global 
environment. 
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